Add XML invalid chars filter instead punctuation interrupt #136
Conversation
Can you describe me your pipeline configuration that feeds into this The point is that the JPMML-SparkML library is following the PMML specification when deciding what can and what cannot be allowed. Punctuation chars can be significant (depending on the text tokenization mode), so they cannot be enabled/disabled at will.
Care to provide an example about this behaviour? It should be the case that the JPMML-SparkML is populating an You're basically claiming that the JAXB marshaller is somehow misbehaving (eg. not encoding/escaping some characters). |
|
In this example I use patched for(int i = 0; i < vocabulary.length; i++){
String term = vocabulary[i];
if(TermUtil.hasPunctuation(term)){
result.add(new TermFeature(encoder, defineFunction, documentFeature, "-"));
} else {
result.add(new TermFeature(encoder, defineFunction, documentFeature, term));
}
}I put this jars into pyspark jars directory ( from pyspark.sql import SparkSession
from pyspark.ml import PipelineModel
from pyspark2pmml import PMMLBuilder
spark = SparkSession.builder.master('local[*]').getOrCreate()
pm = PipelineModel.load('./model.bin')
pmmlBuilder = PMMLBuilder(spark.sparkContext, spark.createDataFrame([('', '')], schema=['lang', 'content']), pm)
pmmlBuilder.buildFile('./model.pmml')(remove .zip suffix from .z01 , z02. required for upload) Output model pmml model will contains invalid xml: import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import java.io.File;
public class Main {
public static void main(String[] args) throws Exception {
DocumentBuilder parser = DocumentBuilderFactory.newInstance().newDocumentBuilder();
parser.parse(new File("./model.pmml"));
}
}(remove .zip suffix from .z01 , z02. required for upload) In vim this parts of file (
|
I try to export models that detect programming languages from input strings. That means I need special symbols like
{or:but current implementations don't allow this. Moreover it allows me to create invalid xml documents when I have invalid XML Characters. I suggest replacing the current implementation that doesn't allow exporting models with punctuated symbols with a new implementation that filters invalid xml chars.